Method and system for measuring user engagement with content items

ABSTRACT

Method, system, and programs for measuring user engagement with content items. In one example, a query is received. A presentation of a content item related to the query is provided on a user interface. A user activity (e.g., related to manipulation of the content item) is determined. A score associated with the content item is determined based on the user activity. Information related to user engagement with the content item is generated based on the score.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is related to a U.S. patent application havingAttorney Docketing No. 022994-0431373, filed on even date, entitledMETHOD AND SYSTEM FOR MEASURING USER ENGAGEMENT WITH CONTENT ITEMS,which is incorporated herein by reference in its entireties.

BACKGROUND

1. Technical Field

The present teaching relates to methods, systems and programming formeasuring user engagement. Particularly, the present teaching isdirected to methods, systems, and programming for measuring userengagement with content items.

2. Discussion of Technical Background

The Internet has made it possible for a user to electronically accessvirtually any content at any time and from any location. With theexplosion of information, it has become increasingly important toprovide users with information that is relevant to the user. Further, asusers of today's society rely on the Internet as their source ofinformation, entertainment, and/or social connections, e.g., news,social interaction, movies, music, etc., it is critical to provide userswith information they find valuable.

Efforts have been made to attempt to enable users to readily accessrelevant content. As an example, there are systems that identify users'interests based on observations made on users' interactions withcontent. In the context of search, for instance, observations regardinguser engagement with search results are typically facilitated viaclick-thru measurements. In particular, a system determines that acontent item has been accessed by a user when the user “clicks” a searchresult link to access the content item as a result of the selected linkcontaining a URL that identifies the accessed content item. As such, bymonitoring which search result links are clicked by users, the systemcan determine which content items are accessed by users and, thus,determine which content items (or their associated search result links)are more interesting to the users overall and/or on a query basis. Suchdeterminations may then be used to personalize the content or the searchresults links that are provided to users during subsequent queries orother user activities.

However, in the context of mobile, a list of search result links may notbe as practical. When approaches other than the traditional list ofsearch result links are utilized to enable users to access content itemsrelated to a query, analysis of user engagement based on click-thrumeasurements may not be an appropriate option. Thus, there is a need formeasuring user engagement with content items without, for example,necessarily relying solely on traditional click-thru measurements.

SUMMARY

The teachings disclosed herein relate to methods, systems, andprogramming for measuring user engagement. More particularly, thepresent teaching relates to methods, systems, and programming formeasuring user engagement with content items.

In one example, a method, implemented on at least one machine each ofwhich has at least one processor, storage, and a communication platformconnected to a network for measuring user engagement, is disclosed. Aquery is received. A presentation of a content item related to the queryis provided on a user interface. A user activity (e.g., related tomanipulation of the content item) is determined. A score associated withthe content item is determined based on the user activity. Informationrelated to user engagement with the content item is generated based onthe score.

In a different example, a system for measuring user engagement isdisclosed. The system includes a search engine system and an engagementassessment system. The engagement assessment system includes a useractivity analysis unit, a user engagement evaluation unit, and a userengagement report generation unit. The search engine system isconfigured to receive a query and to provide, on a user interface, apresentation of a content item related to the query. The user activityanalysis unit is configured to determine a user activity (e.g., relatedto manipulation of the content item). The user engagement evaluationunit is configured to determine a score associated with the content itembased on the user activity. The user engagement report generation unitis configured to generate information related to user engagement withthe content item based on the score.

Other concepts relate to software for measuring user engagement. Asoftware product, in accord with this concept, includes at least onemachine-readable non-transitory medium and information carried by themedium. The information carried by the medium may be executable programcode data regarding parameters in association with a request oroperational parameters, such as information related to a user, arequest, or a social group, etc.

In one example, a machine readable and non-transitory medium hasinformation recorded thereon for measuring user engagement, wherein theinformation, when read by the machine, causes the machine to perform aseries of steps. A query is received. A presentation of a content itemrelated to the query is provided on a user interface. A user activity(e.g., related to manipulation of the content item) is determined. Ascore associated with the content item is determined based on the useractivity. Information related to user engagement with the content itemis generated based on the score.

BRIEF DESCRIPTION OF THE DRAWINGS

The methods, systems and/or programming described herein are furtherdescribed in terms of exemplary embodiments. These exemplary embodimentsare described in detail with reference to the drawings. Theseembodiments are non-limiting exemplary embodiments, in which likereference numerals represent similar structures throughout the severalviews of the drawings, and wherein:

FIGS. 1-2 depict exemplary embodiments of a networked environment inwhich user engagement measurement is applied, according to differentembodiments of the present teaching;

FIG. 3 is a high level exemplary system diagram of an engagementassessment system, according to an embodiment of the present teaching;

FIG. 4 is a high level exemplary diagram of different user activities,according to an embodiment of the present teaching;

FIG. 5 is a high level exemplary diagram of different manipulationactivities, according to an embodiment of the present teaching;

FIG. 6 is a high level exemplary diagram of measuring time betweenevents, according to an embodiment of the present teaching;

FIG. 7 is a high level exemplary system diagram of a search enginesystem and an engagement assessment system, according to an embodimentof the present teaching;

FIG. 8 is a high level exemplary system diagram of a user engagementevaluation unit, according to an embodiment of the present teaching;

FIG. 9 is a flowchart of an exemplary process of measuring userengagement with content items, according to an embodiment of the presentteaching;

FIGS. 10-14 illustrate exemplary user activities with respect to contentitems, according to different embodiments of the present teaching;

FIG. 15 depicts a general mobile device architecture on which thepresent teaching can be implemented; and

FIG. 16 depicts a general computer architecture on which the presentteaching can be implemented.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

The present teaching relates to measuring user engagement with contentitems. In various embodiments, a presentation of a content item isprovided on a user interface to a user in response to receipt of a queryfrom the user. The presentation of the content item is provided, forexample, without providing an intermediate set of results related to thequery after the receipt of the query and before the presentation of thecontent item is provided to the user. In some embodiments, the contentitem is an information card. Other content items can, for example, bepresented as information in respective portions of the information card.In other embodiments, the content item comprises at least one of awebpage, a video, an image, an audio, a document, or other content item.User activities related to the content item is monitored, and userengagement information is generated based on the monitored useractivities (e.g., a report comprising information about content itemsand their associated user engagement scores).

In an embodiment, the user activities comprises a user activity relatedto manipulation of the content item, a user activity related tomanipulation of the presentation of the content item, a user activityrelated to manipulation of metadata associated with the content item, orother manipulated-related user activity. One or more scores associatedwith the content item are determined based on the manipulationactivities. Information related to user engagement with the content itemis then determined based on the scores. As an example, manipulation ofthe content item may comprise modification of an instance of the contentitem and storage of the modified instance such that a subsequentpresentation of the content item to the user will comprise apresentation of the modified instance of the content item. As anotherexample, manipulation of the presentation of the content item maycomprise modification of the presentation of the content item such thata subsequent presentation of the content item to the user will comprisea presentation of the non-modified instance of the content item. As yetanother example, manipulation of the presentation of the content itemmay comprise manipulating which portions of the content item arecurrently in view (e.g., scrolling so that certain portions of thecontent item are in view while other portions are not in view). As afurther example, manipulation of metadata associated with the contentitem may comprise modification of the metadata associated with thecontent item. In one use case, when a user “likes” or “dislikes” acontent item that is presented responsive to a query, the “like” or“dislike” may be stored as metadata associated with the content item,thereby modifying the metadata associated with the content item.

In another embodiment, amounts of time between a time at which thepresentation of the content item is provided to the user and times atwhich various user activities related to the content item occur aredetermined. In yet another embodiment, amounts of time between the timesat which various user activities related to the content item occur aredetermined. One or more scores associated with the content item aredetermined based on the amounts of time. Information related to userengagement with the content item is then determined based on the scores.

In a further embodiment, various user activities that occur during apresentation of one or more content items of a set of content itemsrelated to the query is determined. One or more scores associated withthe set of content items are determined based on the various useractivities. In some embodiments, the scores are determined based on anamount of time between times at which various ones of the useractivities occurred. Information related to user engagement with the setof content items is then determined based on the scores.

As discussed, in the context of mobile or other similar environments, alist of search result links may not be as practical. When approachesother than the traditional list of search result links are utilized toenable users to access content items related to a query, analysis ofuser engagement based solely on click-thru measurements may not be anappropriate option. For example, search results can be presented as“cards” that are loaded with content relevant to a user query, reducingthe need for a user to click/tap on a link to access an external orthird party site that comprise the same content. As such, it isimportant not to rely solely on click-thru activities in such scenarios,and to measure user engagement based on other user activities, such asscrolling vertically through information, swiping horizontally through acarousel of information, pinches, zooms, rotations, dismissals,collapses, external application selection actions related to theinformation cards, etc.

FIG. 1 is a high level depiction of an exemplary system 100 in which anengagement assessment system 140 is deployed to measure user engagementwith content items, according to an embodiment of the present teaching.The exemplary system 100 includes users 110, a network 120, a searchengine system 130, content sources 160, an engagement assessment system140, content sources 160, or other components. The network 120 in system100 can be a single network or a combination of different networks. Forexample, a network can be a local area network (LAN), a wide areanetwork (WAN), a public network, a private network, a proprietarynetwork, a Public Telephone Switched Network (PSTN), the Internet, awireless network, a virtual network, or any combination thereof. Anetwork may also include various network access points, e.g., wired orwireless access points such as base stations or Internet exchange points120 a-120 n, through which a data source may connect to the network inorder to transmit information via the network.

Users 110 may be of different types such as users connected to thenetwork via desktop connections (110-d), users connecting to the networkvia wireless connections such as through a laptop (110-c), a handhelddevice (110 a), or a built-in device in a motor vehicle (110 b). A usermay submit a query to the search engine system 130 via the network 120and receive a query result from the search engine system 130 through thenetwork 120. In some embodiments, the user may be provided with apresentation of content items without first being provided with anintermediate set of results related to the query after the submission ofthe query and before the presentation of the content items. For example,the presentation of the content items may be provided to the userwithout first presenting the user with a list of search result links andrequiring the user to select (e.g., by clicking, tapping, etc.) one ofthe presented search result links to be provided with a presentation ofone of the content items.

In some embodiments, a browser (or other application) at a user devicemonitors activities at the user device, such as when a presentation of acontent item is loaded on the browser, when certain user activities(e.g., actions, in-actions, etc.) related to the content item occurs,etc. Responsive to the monitoring, the browser (or other application)generates information regarding the user activities, informationregarding the timing of the presentation or the user activities, orother information. Subsequently, the generated information istransmitted to one or more servers (e.g., a server comprising the searchengine system 130, the engagement assessment system 140, or both). Theengagement assessment system 140 can then determine the user activitiesthat occurred at the user device, the amounts of time between the timeat which the presentation of the content item is provided to the userand the times at which certain user activities occurred, the amounts oftime between the times at which various user activities occurred, orother information.

The content sources 160 include multiple content sources 160 a, 160 b, .. . 160 n. A content source may correspond to a web page hostcorresponding to an entity, whether an individual, a business, or anorganization such as USPTO.gov, a content provider such as cnn.com andYahoo.com, or a content feed source such as Twitter or blogs. Both thesearch engine system 130 and the engagement assessment system 140 mayaccess information from any of the content sources 160 a, 160 b, . . . ,160 n and rely on such information to respond to a query (e.g., thesearch engine system 130 identifies content related to keywords in thequery and returns the result to a user).

FIG. 2 is a high level depiction of an exemplary system 200 in which theengagement assessment system 140 is deployed to measure user engagementwith content items, according to an embodiment of the present teaching.The exemplary networked environment 200 in this embodiment is similar tothe exemplary networked environment 100 in FIG. 1, except that theengagement assessment system 140 in this embodiment connects to thenetwork 120 via the search engine system 130. For example, theengagement assessment system 140 may serve as a backend system of thesearch engine system 130.

FIG. 3 depicts the inputs and output of the engagement assessment system140, according to an embodiment of the present teaching. As shown, theengagement assessment system 140 receives user activity information froma user device as input. The user activity information is, for example,received in real-time as the user activities are occurring, on aperiodic basis, based on a schedule, etc. The user activity informationcomprises information regarding the content items with which usersinteracted, information regarding the types of user activities thatoccurred, information that can be used to identify users (e.g., withoutpersonally identifiable information) with which the user activities areassociated, information regarding the times at which the user activitiesoccurred, information regarding the times at which the presentations ofthe content items are provided to the users, or other information. Byanalyzing the user activity information, the engagement assessmentsystem 140 can generate user engagement information that can be utilizedto personalize content items for users (e.g., that are provided inresponse to subsequent queries), organize the content items for users,provide recommendations to users, or provide other benefits.

FIG. 4 is a high level exemplary diagram of different user activities,according to an embodiment of the present teaching. As illustrated inFIG. 4, users activities may comprise user activities related tomanipulation, user activities related to dwell time, or other useractivities. In an embodiment, manipulation activities comprises useractivities related to manipulation of a content item, user activitiesrelated to manipulation of a presentation of the content item, useractivities related to manipulation of metadata associated with thecontent item, or other manipulation activities. As an example,manipulation of the content item may comprise modification of aninstance of the content item and storage of the modified instance suchthat a subsequent presentation of the content item to the user willcomprise a presentation of the modified instance of the content item. Asshown in FIG. 5, for instance, manipulation of the content item includesrotations of the content item (or portions thereof), resizing of thecontent item (or portions thereof), adding portions to the content item,moving portions within the content item, removing portions from thecontent item, or other modification.

As another example, manipulation of the presentation of the content itemmay comprise modification of the presentation of the content item (e.g.,such that a subsequent presentation of the content item to the user willcomprise a presentation of the non-modified instance of the contentitem), control of which portions of the content item are currently inview (e.g., scrolling so that certain portions of the content item arein view while other portions are not in view), removal of thepresentation of the content item, or other manipulation. As shown inFIG. 5, manipulation of the presentation of the content item includesrotations of the presentation (or portions thereof), resizing of thepresentation (or portions thereof) adding portions to the presentation,moving portions within the presentation, removing portions from thepresentation, scrolling through portions of the content item, removingthe presentation, or other manipulation.

As yet another example, manipulation of metadata associated with thecontent item may comprise modification of the metadata associated withthe content item. As depicted in FIG. 5, manipulation of the metadataassociated with the content item includes modifications of metadataindicating ratings (e.g., “likes,” “dislikes,” numerical ratings, etc.),metadata indicating share (e.g., to/by whom the content item has beenshared, number of shares, etc.), metadata indicating favorites (e.g., bywhom the content item has been favorite, number of users that favoritethe content item, etc.), or other metadata. In one use case, when a user“likes” or “dislikes” a content item that is presented responsive to aquery, the “like” or “dislike” may be stored as metadata associated withthe content item, thereby modifying the metadata associated with thecontent item.

Referring back to FIG. 4, user activities related to dwell time maycomprise user activities that last for a period of time, such asremaining in a particular state for a period of time. As shown in FIG.6, after a presentation of an information card is provided to a user attime T0 (610), the user did not interact with the information card untiltime T1 (620) at which the user performed a first user action withrespect to the information card. As such, the user's remained inactivewith respect to interaction with the information card for a time periodt1. The time period t1 may then be utilized to determine the level ofinterest of the user with the presentation of the information card. Asan example, if the time period t1 is short, then the user is more likelyto have been interested in the information card because the user quicklyinteracted with the information card. On the other hand, if the timeperiod t1 is long, then the user is less likely to have been interestedin the information card. Of course, other factors, such as the nature ofcontent presented on the information card or the nature of the firstuser action, may be considered to obtain a better estimate of the levelof interest of the user with the information card (e.g., an informationcard with a considerable amount of text may result in the user readingthe text currently in view before scrolling to other portions of theinformation card).

In addition, as depicted in FIG. 6, the user has been recorded to havespent a time period t2 with the presentation of the information card(e.g., from time T0 until time T4 during which events 610, 620, 630,640, and 650 occur). As with the time period t1, the time period t2 maybe utilized to determine the level of interest of the user with theinformation card. For example, if the time period t2 is short, then theuser is less likely to have been interested in the information cardbecause the user removed the presentation of the information card (e.g.,removal from view, discarding of the information card, etc.) in a shortperiod of time. On the other hand, if the time period t2 is long, thenthe user is more likely to have been interested in the information card.Again, other factors may be considered to obtain a better estimate ofthe level of interest of the user with the information card (e.g., theinformation card may only comprise a small amount of content).

Moreover, as illustrated in FIG. 6, the amount of time between useractivities (e.g., between user actions, between a user action and userinaction, etc.) may be considered to determine the level of interest ofthe user with the information card. Although FIG. 6 depicts a timeperiod t3 to describe the amount of time between two user actions withrespect to the information card, time periods between a user action anda user inaction may also be considered to determine the level ofinterest of users with content items (e.g., the information card).

In one use case, for example, a scroll action is performed during apresentation of an information card. If an amount of time between thestart time of the scroll action and the end time of the scroll action(e.g., amount of time between related action and inaction) is a shortperiod of time, but the length of the portions of the information cardthat user scrolled through is long (e.g., very fast scroll), it may bedeemed unlikely that the user actually read the information on thescrolled-through portions of the information card. As such, it is lesslikely that the user was interested in the portions of the informationcard that the user scrolled through.

In another use case, a second scroll action is performed during thepresentation of the information card. If an amount of time between thestart time of the second scroll action and the end time of the secondscroll action is long, but the length of the portions of the informationcard that the user scrolled through is short (e.g., very slow scroll),it may be deemed likely that the user carefully read thescrolled-through portions of the information card. Thus, it is morelikely that the user was interested in the portions of the informationcard that the user scrolled through.

FIG. 7 depicts a high level exemplary system diagram of the searchengine system 130 and the engagement assessment system 140, according toan embodiment of the present teaching. As shown, users 710 submitqueries to the search engine system 130, and the search engine system130 provides the users 710 with presentations of content items relatedto the submitted queries. Information regarding user activitiesoccurring at user devices of the users 710 is received at the engagementassessment system 140, which then processes the user activityinformation to generate user engagement information.

As depicted in FIG. 7, the engagement assessment system 140 comprises auser activity analysis unit 720, a user engagement evaluation unit 730,and a user engagement report generation unit 750. As an example, whenthe user activity information is received at the engagement assessmentsystem 140, the user activity analysis unit 720 processes the useractivity information to determine the content items with which the usersinteracted, what user activities occurred at the user devices of theusers 710 with respect to the content items, which users interacted withwhich content items, the times at which the user activities occurred,the times at which the presentations of the content items are providedto the users, etc.

In addition, in some embodiments, the user activity analysis unit 720determines, based on the user activity information, the nature of theuser activities, such as the type of each user activity, the amount oftime between pairs of user activities, or the amount of time between auser activity and another event (e.g., start or end of a presentation ofa content item, start or end of a presentation of a set of content item,etc.). Such determinations are then provided to the user engagementevaluation unit 730 to determine scores 740 associated with the useractivity and/or the content items.

As discussed, in one use case, if an amount of time between the starttime of a scroll action on an information card and the end time of thescroll action (e.g., amount of time between related action and inaction)is a short time period, but the length of the portions of theinformation card that user scrolled through is long (e.g., very fastscroll), it may be deemed unlikely that the user actually read theinformation on the scrolled-through portions of the information card.Thus, the user engagement evaluation unit 730 may associate a low scorewith the portions of the content item that the user scrolled through.

In another use case, if an amount of time between the start time of ascroll action on an information card and the end time of the scrollaction is long, but the length of the portions of the information cardthat the user scrolled through is short (e.g., very slow scroll), it maybe deemed likely that the user carefully read the scrolled-throughportions of the information card. Therefore, the user engagementevaluation unit 730 may associate a high score with the portions of thecontent item that the user scrolled through.

In a further user case, the low score associated with the firstscrolled-through portions and the high score associated with the secondscrolled-through portions may be processed (along with other scoresassociated with the information card) to determine a combined scoreassociated with the information card. It is understood that, althoughexemplary scoring metrics are described herein, they are by ways ofexample rather than limitation. Any other appropriate scoring metricscan be utilized to determine scores associated with content items withinthe scope of the present teaching. For example, other scoring metricsmay comprise the amount of time that a user dwells on a card or set ofcards, a number of cards browsed during presentation of a set of cards,the rate of certain user activities (or types of user activities) thatoccur during presentation of a card or a set of cards, the number ofcards removed during a presentation of a set of cards (e.g.,“discarding” cards), the number of cards saved by the user duringpresentation of a set of cards (e.g., bookmarked for later viewing), theratings given to a set of cards by the user (e.g., likes, dislikes,numerical rating, etc.), the number of cards that are shared by the userwith other users during presentation of a set of cards, etc.

As illustrated in FIG. 8, in an embodiment, the user engagementevaluation unit 730 comprises an activity scoring determination unit, ascoring database 820, and a score-based engagement estimation unit 830.As discussed, in some embodiments, the user activity analysis unit 720determines the nature of user activities, such as the type of each useractivity, the amount of time between pairs of user activities, or theamount of time between a user activity and another event (e.g., start orend of a presentation of a content item, start or end of a presentationof a set of content item, etc.). The user activity analysis unit 720then categorizes/classifies the user activities and provides informationregarding the categorized/classified user activities to the userengagement evaluation unit 730.

As shown, the activity scoring determination unit 810 processes theclassified activity information to obtain scoring information from thescoring database 820. In some embodiments, the scoring database 820comprises information regarding scores to be assigned to the useractivities based on the user activity types, the amount of time betweenpairs of user activities, or the amount of time between a user activityand another event. The scoring database 820 may further compriseinformation regarding weights to be assigned to scores based oninformation surround the scores such as the user activity types, theamount of time between pairs of user activities, or the amount of timebetween a user activity and another event. The scoring determinationunit 810 obtains the scores and associated weights from the scoringdatabase 820 based on the classified activity information. The scoringdetermination unit 810 then provides the scores and associated weightsto the score-based engagement estimation unit 830.

The score-based engagement estimation unit 830 utilizes the scores andthe weights to estimate user engagement with a content item or a set ofcontent items related to a query. As depicted in FIG. 8, the score-basedengagement estimation unit 830 estimates the user engagement in the formof user engagement scores 740. For example, the score-based engagementestimation unit 830 computes one or more user engagement scores for acontent item based on scores and weights associated with user activitiesrelated to the content item.

In some embodiments, referring back to FIG. 7, the user engagementscores 740 is provided to user engagement report generation unit 750.The user engagement report generation unit 750 utilizes the userengagement scores 740 to generates information related to userengagement with a content item or a set of content items in the form ofa user engagement assessment. In an embodiment, the user engagementreport generation unit 750 obtains user engagement information from theengagement database 760, and utilizes the user engagement informationfrom the engagement database 760 and the user engagement scores 740 toupdate the user engagement information. The updated user engagementinformation may then be utilized by the search engine system 130 toselect content items to provide to users in response to subsequentqueries.

As am example, if the user engagement scores 740 are associated with aparticular content item, the user engagement report generation unit 750may obtain prior user engagement information associated with the contentitem or a similar content item. The user engagement report generationunit 750 then processes the prior user engagement information and theuser engagement scores 740 to generate updated/supplemental userengagement information for the content item or content items that aresimilar to the content item. The updated/supplemental user engagementinformation may subsequently be utilized by the search engine system 130to determine whether and/or how to present the content item or similarcontent items (e.g., if the content items are relevant to a terms of aquery, the search engine system 130 determines whether and/or how topresent the content items based on the user engagement information).

FIG. 9 is a flowchart of an exemplary process of measuring userengagement with content items, according to an embodiment of the presentteaching. As shown, a query is received at 910. In response to thereceived query, a search for content items may be performed based on thequery at 920. As such, in response to the search, one or more contentitems related to the query are obtained. A presentation of at least oneof the obtained content items is provided on a user interface at 930. Asindicated, in some embodiments, the presentation of the content item maybe provided to a user without providing an intermediate set of resultsrelated to the query after the receipt of the query and before thepresentation of the content item. For example, the presentation of thecontent item may be provided to the user without first presenting theuser with a list of search result links and requiring the user to select(e.g., by clicking, tapping, etc.) one of the presented search resultlinks to be provided with the presentation of the content item.

At 940, user activities are determined. As discussed, and shown in FIG.4, users activities may comprise user activities related tomanipulation, user activities related to dwell time, or other useractivities. In one use case, one or more of the determined useractivities comprises one or more manipulation activities. In someembodiments, a manipulation activity (or other user activity) isinitially detected at a user device. The user device then transmitsinformation indicating the detection of the user activity to one or moreservers (e.g., a server comprising the search engine system 130, theengagement assessment system 140, or both). The engagement assessmentsystem 140, for instance, determines that the user activity occurredbased on the information from the user device that indicates thedetection of the user activity.

As an example, a user performs a user action configured to modify atleast a portion of the content item (e.g., rotate command, move command,crop command, delete command, etc.). In response, an application at theuser device (e.g., browser or other application) performs themodification indicated by the user action on a local instance of thecontent item, and then transmits the modified instance to one or moreservers comprising the search engine system 130 and the engagementassessment system 140 along with information indicating the detecteduser action. Upon receipt of the modified instance, the search enginesystem 130 stores the modified instance such that a subsequentpresentation of the content item to the user will comprise apresentation of the modified instance of the content item. Theengagement assessment system 140 will also determine that the useraction occurred at the user device with respect to the content itembased on the information from the user device indicating the detecteduser action.

As another example, when a user performs a user action configured tomodify at least a portion of the content item, the user device transmitsinformation indicating the detected user action to one or more serversto cause an instance of the content item at one of the servers to bemodified based on the detected user action. Upon receipt of theinformation indicating the detected user action, the search enginesystem 130 modifies the instance of the content item and stores themodified instance such that a subsequent presentation of the contentitem to the user will comprise a presentation of the modified instanceof the content item. The information indicating the detected user actionmay also be utilized by the engagement assessment system 140 todetermine that the user action occurred at the user device with respectto the content item.

At 950, the nature of each of the user activities is determined. As anexample, the engagement assessment system 140 determines the type ofeach user activity, the amount of time between pairs of user activities,or the amount of time between a user activity and another event (e.g.,start or end of a presentation of a content item, start or end of apresentation of a set of content item, etc.). In one use case, forinstance, the engagement assessment system 140 obtains (e.g., from auser device) user activity information that indicates user activitiesthat occurred at the user device, and processes the user activityinformation to determine the nature of each of the user activities.

At 960, a score associated with each user activity and/or with thecontent item is determined based on the nature of each user activity. Inone use case, user activities that generally require more thought fromthe user (e.g., modifying a content item or presentation thereof) mayresult in higher scores for the content item than user activities thatrequire less thought from the user (e.g., scrolling, removing from view,etc.). In another use case, user activities that correspond to explicitindications of interest (e.g., rating a content item, sharing a contentitem, etc.) may result in scores that directly correlate with theexplicit indications. For example, if a user rates the content item a“5” on a scale of 1-5, and a score for the content item may be “10” on ascale of 1-10 before other factors are considered. In yet another usecase, a score for the content item may be based on the amount of timebetween the occurrence of a user activity and the occurrence of someother event.

As an example, in one scenario, the content item is an information cardrelated to a user query (e.g., cards 1040 a-1040 n) on which contentrelated to the information card is presented, and the user activity is auser activity that removes the information card from view of the user(e.g., a swipe action that removes the information card from a set ofinformation cards that is presented to the user, a drag and drop actionthat places the information card at the bottom of the set of informationcards, etc.). A score is determined for the content item based on howlong a user spends with the information card before removing theinformation card from view and/or other factors (e.g., an amount ofmanipulated activities that the user performs on the information card).If the user spends a long amount of time with the information cardbefore removing the information card from view, a higher score may beassigned to the information card. On the other hand, if the user spendsa short amount of time with the information card before removing theinformation card from review, a lower score may be assigned to theinformation card.

In another scenario, a score is determined for one or more portions ofthe information card (e.g., content items presented within theinformation card) based on how long a user spends with the portions ofthe information card before scrolling to other portions of theinformation card. If the user spends a long amount of time with certainportions of the information card before scrolling to other portions ofthe information card, a higher score may be assigned to the portions ofthe information card. On the other hand, if the user spends a shortamount of time with the portions of the information card, a lower scoremay be assigned to the portions of the information card. In someembodiments, the scores assigned to information cards (or portionsthereof) is subsequently used to determine the relevancy of theinformation card to the user query that produced the information card,the level of interest of the user in the information card, or otherinformation. Such information can then be utilized to personalize theset of information cards and/or the content of the information cardsthat are presented to the user (or other users) in subsequent relatedqueries.

At 970, user engagement with the content item is estimated based on thescore. As indicated, in an embodiment, the estimation of the userengagement is used for subsequent queries (e.g., that are the same orsimilar to the current query) to select content item to present to usersthat submitted the queries (e.g., for subsequent queries submitted bythe same user that performed the user activities from which theestimation is based, for subsequent queries submitted by users similarto that user, or for subsequent queries submitted by other users). Inanother embodiment, as shown in FIG. 9, the estimation of the userengagement is used to search for further content items to be presentedto the user that submitted the current query. As such, as the user isinteracting with a set of content items, new content items may bedynamically added to the set of content items based on the estimation ofthe user engagement with the current set of content items. In yetanother embodiment, the estimation of the user engagement is used todynamically remove content items from the current set of content itemsbased on a determination that those content items would not be ofinterest to the user.

In some embodiments, information related to user engagement with a setof content items (e.g., as a whole) is generated based on scoresassociated with individual content items, scores associated with the setof content items, or other scores. As an example, with respect to FIG.9, a set of content items may be obtained in response to receiving aquery at 910 and 920. A presentation of at least one content item of theset of content items is provided to a user at 930. At 940, 950, and 960,user activities and their respective nature are determined, and a score(or scores) associated with the presented content item and/or each useractivity are determined based on the nature of each user activity. Inaddition, user engagement with the set of content items (e.g., as awhole) is then estimated based on the score(s) associated with thepresented content item (or user activities thereof) along with otherscores associated with other content items or with the set of contentitems (or user activities thereof).

FIGS. 10-14 illustrate exemplary user activities with respect to contentitems, according to different embodiments of the present teaching. Forexample, FIG. 10 illustrates a user interface 1010 on a mobile device1020 after a user has submitted query terms in query input area 1030. Inresponse to the submission of the query terms, a stack of informationcards 1040 a-1040 n is presented to the user on the user interface 1010.As shown, in some embodiments, the presentation of the information cardsis provided to a user without providing an intermediate set of resultsrelated to the query after the receipt of the query and before thepresentation of the information cards. The presentation of theinformation cards is, for example, provided to the user without firstpresenting the user with a list of search result links and requiring theuser to select (e.g., by clicking, tapping, etc.) one of the presentedsearch result links to be provided with the presentation of the contentitems. As depicted, the information card 1040 a is presented on top ofthe other information cards 1040 such that content of the informationcard 1040 a (e.g., within portions of the information card 1040 a) is inview on the user interface 1010. In some embodiments, the user can viewor otherwise access the content of the other information cards byswiping away the information card 1040 a, dragging the information card1040 a to another position within the stack of information cards 1040,selecting another one of the information cards 1040, etc. In someembodiments, each of the information cards 1040 may correspond to arespective domain (e.g., weather, restaurants, movies, music,navigation, calendar, etc.). Viewing or otherwise accessing the contentof other information cards may thus allow a user to view or otherwiseaccess the content of information pertaining to other domains.

FIG. 11 illustrates a rotate action being performed with respect tocontent in Portion 1 of the information card 1040 a. In an embodiment,the depicted rotate action is an action that triggers a modification ofthe information card 1040 a (or the Portion 1 content) such that aninstance of the information card 1040 a (or the Portion 1 content) ismodified and then stored for subsequent presentation of the informationcard 1040 a. In one use case, for example, the modified version of theinformation card 1040 a (or the rotated Portion 1 content) is presentedto the user (instead of the original version of the information card1040 a) in response to a subsequent user query. In another embodiment,the depicted rotate action is an action that triggers a modification ofthe presentation of the information card 1040 a (or the Portion 1content) such that a subsequent presentation of the information card1040 comprises the original version of the information card 1040 a (orthe non-rotated Portion 1 content).

FIG. 12 illustrates a removal action being performed with respect tocontent in Portion 2 of the information card 1040 a. In an embodiment,the depicted removal action is an action that triggers a modification ofthe information card 1040 a (or the Portion 1 content) such that aninstance of the information card 1040 a (or the Portion 1 content) ismodified and then stored for subsequent presentation of the informationcard 1040 a. In one scenario, for example, the modified version of theinformation card 1040 a (e.g., the information card 1040 a without thePortion 2 content) is presented to the user (instead of the originalversion of the information card 1040 a) in response to a subsequent userquery. In another embodiment, the depicted removal action is an actionthat triggers a modification of the presentation of the information card1040 a (or the Portion 1 content) such that a subsequent presentation ofthe information card 1040 comprises the original version of theinformation card 1040 a (e.g., the information card 1040 a with thePortion 2 content).

FIG. 13 illustrates a removal action being performed to remove theinformation card 1040 a from the presentation of the stack ofinformation cards 1040. The removal action may, for example, comprise aswipe action, a drag-and-drop action, or other action that triggers theremoval of the information card 1040 a from the presentation of thestack of information cards 1040. After the removal action in FIG. 13 isperformed on the information card 1040, the information card 1040 b ispresented to the user on top of the stack of information cards 1040 b-nsuch that the contents of the information card 1040 b (e.g., withinportions of the information card 1040 b) is in view on the userinterface 1010.

FIG. 14 illustrates a scroll action being performed with respect to theinformation card 1040 b. As shown, the user performs the scroll actionon information card 1040 b such that the content of Portions 2, 3, and 4of the information card 1040 b is in view on the user interface 1010 (asopposed to the content of Portions 1 and 2 of the information card 1040b). It is understood that, although exemplary user activities aredescribed herein (e.g., with respect to FIGS. 10-14), they are by waysof example rather than limitation. Any other appropriate user activitycan be performed, monitored, detected, and/or utilized to provideinformation related to user engagement with content items within thescope of the present teaching. Other examples comprise moving a portionof a content item from one position of the content item to anotherposition of the content item, adding content to the content item, etc.

In some embodiments, each of the user activities described with respectto FIGS. 10-14 are monitored so that user engagement with theinformation cards 1040 may be analyzed based on those user activities.In an embodiment, scores are determined for each information card 1040based on a user's activities related to the respective information card1040. As an example, scores are determined for each information card1040 based on the amount of time that the user spent on the respectiveinformation card 1040 before removing the information card 1040 fromview, the rate at which the user scrolls through portions of therespective information card 1040, whether the user shares the respectiveinformation card 1040 (or portions thereof) with other users, how theuser rates the respective information card 1040 (e.g., likes, dislikes,numerical rating, etc.), or other parameters related to user activity.

In another embodiment, the scores associated with each of theinformation cards 1040 are analyzed to determine one or more scoresassociated with the overall set of information cards 1040. As anexample, a score may be determined for the overall set of informationcards 1040 with regard to how relevant the user perceives the set ofinformation cards 1040 to be with respect to the query (e.g., the queryterms shown in FIGS. 10-14). As another example, a score may bedetermined for the overall set of information cards 1040 with regard tothe level of interest the user has with the overall set of informationcards 1040.

In another embodiment, additional (or alternative) scores are determinedfor the overall set of information cards 1040 based on metrics, such asthe total amount of time that the user spend on the overall set ofinformation cards 1040, a number of the information cards 1040 that theuser browsed during presentation of the overall set, the rate of certainuser activities (or types of user activities) that occur duringpresentation of the overall set, the number of cards removed duringpresentation of the overall set, the number of cards saved by the userduring presentation of the overall set (e.g., bookmarked for laterviewing), the ratings given to the information cards by the user duringpresentation of the overall set (e.g., likes, dislikes, numericalrating, etc.), the number of information cards that are shared by theuser with other users during presentation of the overall set, etc.

FIG. 15 depicts a general mobile device architecture on which thepresent teaching can be implemented. In this example, the user device onwhich content item (with which user engagement is measured) is presentedis a mobile device 1500, including but is not limited to, a smart phone,a tablet, a music player, a handled gaming console, a global positioningsystem (GPS) receiver. The mobile device 1500 in this example includesone or more central processing units (CPUs) 1502, one or more graphicprocessing units (GPUs) 1504, a display 1506, a memory 1508, acommunication platform 1510, such as a wireless communication module,storage 1512, and one or more input/output (I/O) devices 1514. Any othersuitable component, such as but not limited to a system bus or acontroller (not shown), may also be included in the mobile device 1500.As shown in FIG. 15, a mobile operating system 1516, e.g., iOS, Android,Windows Phone, etc., and one or more applications 1518 may be loadedinto the memory 1508 from the storage 1512 in order to be executed bythe CPU 1502. The applications 1518 may include a browser or any othersuitable mobile apps for receiving user queries, providing apresentation of a content item related to the query, detecting useractivities related to the content item, or performing other operations.

To implement the present teaching, computer hardware platforms may beused as the hardware platform(s) for one or more of the elementsdescribed herein. The hardware elements, operating systems, andprogramming languages of such computers are conventional in nature, andit is presumed that those skilled in the art are adequately familiartherewith to adapt those technologies to implement the processingessentially as described herein. A computer with user interface elementsmay be used to implement a personal computer (PC) or other type of workstation or terminal device, although a computer may also act as a serverif appropriately programmed. It is believed that those skilled in theart are familiar with the structure, programming, and general operationof such computer equipment and as a result the drawings should beself-explanatory.

FIG. 16 depicts a general computer architecture on which the presentteaching can be implemented and has a functional block diagramillustration of a computer hardware platform that includes userinterface elements. The computer may be a general-purpose computer or aspecial purpose computer. This computer 1600 can be used to implementany components of the user engagement measurement architecture asdescribed herein. Different components of the system in the presentteaching can all be implemented on one or more computers such ascomputer 1600, via its hardware, software program, firmware, or acombination thereof. Although only one such computer is shown, forconvenience, the computer functions relating to the target metricidentification may be implemented in a distributed fashion on a numberof similar platforms, to distribute the processing load.

The computer 1600, for example, includes COM ports 1602 connected to andfrom a network connected thereto to facilitate data communications. Thecomputer 1600 also includes a central processing unit (CPU) 1604, in theform of one or more processors, for executing program instructions. Theexemplary computer platform includes an internal communication bus 1606,program storage and data storage of different forms, e.g., disk 1608,read only memory (ROM) 1610, or random access memory (RAM) 1612, forvarious data files to be processed and/or communicated by the computer,as well as possibly program instructions to be executed by the CPU. Thecomputer 1600 also includes an I/O component 1614, supportinginput/output flows between the computer and other components thereinsuch as user interface elements 1616. The computer 1600 may also receiveprogramming and data via network communications.

Hence, aspects of the method of measuring user engagement, as outlinedabove, may be embodied in programming. Program aspects of the technologymay be thought of as “products” or “articles of manufacture” typicallyin the form of executable code and/or associated data that is carried onor embodied in a type of machine readable medium. Tangiblenon-transitory “storage” type media include any or all of the memory orother storage for the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide storage at any time for thesoftware programming.

All or portions of the software may at times be communicated through anetwork such as the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another. Thus, another typeof media that may bear the software elements includes optical,electrical, and electromagnetic waves, such as used across physicalinterfaces between local devices, through wired and optical landlinenetworks and over various air-links. The physical elements that carrysuch waves, such as wired or wireless links, optical links or the like,also may be considered as media bearing the software. As used herein,unless restricted to tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Hence, a machine readable medium may take many forms, including but notlimited to, a tangible storage medium, a carrier wave medium or physicaltransmission medium. Non-volatile storage media include, for example,optical or magnetic disks, such as any of the storage devices in anycomputer(s) or the like, which may be used to implement the system orany of its components as shown in the drawings. Volatile storage mediainclude dynamic memory, such as a main memory of such a computerplatform. Tangible transmission media include coaxial cables; copperwire and fiber optics, including the wires that form a bus within acomputer system. Carrier-wave transmission media can take the form ofelectric or electromagnetic signals, or acoustic or light waves such asthose generated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer can read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to aprocessor for execution.

Those skilled in the art will recognize that the present teachings areamenable to a variety of modifications and/or enhancements. For example,although the implementation of various components described above may beembodied in a hardware device, it can also be implemented as a softwareonly solution. In addition, the components of the system as disclosedherein can be implemented as a firmware, firmware/software combination,firmware/hardware combination, or a hardware/firmware/softwarecombination.

While the foregoing has described what are considered to be the bestmode and/or other examples, it is understood that various modificationsmay be made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that the teachings may beapplied in numerous applications, only some of which have been describedherein. It is intended by the following claims to claim any and allapplications, modifications and variations that fall within the truescope of the present teachings.

We claim:
 1. A method, implemented on a machine having at least oneprocessor, storage, and a communication platform connected to a network,for measuring user engagement with content items, comprising the stepsof: receiving a query; providing, on a user interface, a presentation ofa content item related to the query; determining a user activity relatedto manipulation of the content item; determining a score associated withthe content item based on the user activity; and generating informationrelated to user engagement with the content item based on the score. 2.The method of claim 1, further comprising: receiving, from a userdevice, information indicating a detection of the user activity at theuser device, wherein the user activity is determined based on theinformation indicating the detection of the user activity at the userdevice.
 3. The method of claim 1, wherein the user activity causes aninstance of the content item to be modified, the method furthercomprising: storing the modified instance of the content item such thata subsequent presentation of the content item comprises a presentationof the modified instance of the content item.
 4. The method of claim 1,wherein the presentation of the content item is provided withoutproviding an intermediate set of results related to the query after thereceipt of the query and before the presentation of the content item. 5.The method of claim 1, further comprising: determining an amount of timebetween a time at which the presentation of the content item is providedon the user interface and a time at which the user activity occurs,wherein at least one of the score or a second score associated with theuser activity is determined based on the amount of time.
 6. The methodof claim 5, wherein the information related to user engagement with thecontent item is generated based on the score and the second score. 7.The method of claim 1, wherein the user activity comprises at least oneof a first operation to rotate a portion of the content item or a secondoperation to move a portion of the content item from one position of thecontent item to another position of the content item.
 8. The method ofclaim 1, wherein the user activity comprises at least one of a firstoperation that is configured to remove a portion of the content itemfrom the content item, a second operation that is configured to modify aportion of the content item, or a third operation that is configured toadd content to the content item.
 9. The method of claim 1, wherein theuser interface comprises a mobile user interface.
 10. The method ofclaim 1, wherein the user activity does not correspond to a click-thruactivity.
 11. A system for measuring user engagement with content items,comprising: a search engine system configured for: receiving a query;and providing, on a user interface, a presentation of a content itemrelated to the query; a user activity analysis unit configured fordetermining a user activity related to manipulation of the content item;a user engagement evaluation unit configured for determining a scoreassociated with the content item based on the user activity; and a userengagement report generation unit for generating information related touser engagement with the content item based on the score.
 12. The systemof claim 11, wherein the user activity analysis unit is furtherconfigured for receiving, from a user device, information indicating adetection of the user activity at the user device, and wherein the useractivity is determined based on the information indicating the detectionof the user activity at the user device.
 13. The system of claim 11,wherein the user activity causes an instance of the content item to bemodified, and wherein the search engine system is further configured forstoring the modified instance of the content item such that a subsequentpresentation of the content item comprises a presentation of themodified instance of the content item.
 14. The system of claim 11,wherein the presentation of the content item is provided withoutproviding an intermediate set of results related to the query after thereceipt of the query and before the presentation of the content item.15. The system of claim 11, wherein the user activity analysis unit isfurther configured for determining an amount of time between a time atwhich the presentation of the content item is provided on the userinterface and a time at which the user activity occurs, and wherein atleast one of the score or a second score associated with the useractivity is determined based on the amount of time.
 16. The system ofclaim 15, wherein the information related to user engagement with thecontent item is generated based on the score and the second score. 17.The system of claim 11, wherein the user activity comprises at least oneof a first operation to rotate a portion of the content item or a secondoperation to move a portion of the content item from one position of thecontent item to another position of the content item.
 18. The system ofclaim 11, wherein the user activity comprises at least one of a firstoperation that is configured to remove a portion of the content itemfrom the content item, a second operation that is configured to modify aportion of the content item, or a third operation that is configured toadd content to the content item.
 19. A machine-readable tangible andnon-transitory medium having information for measuring user engagementswith content items recorded thereon, wherein the information, when readby the machine, causes the machine to perform the following: receiving aquery; providing, on a user interface, a presentation of a content itemrelated to the query; determining a user activity related tomanipulation of the content item; determining a score associated withthe content item based on the user activity; and generating informationrelated to user engagement with the content item based on the score. 20.The medium of claim 19, wherein the presentation of the content item isprovided without providing an intermediate set of results related to thequery after the receipt of the query and before the presentation of thecontent item.